Tooltip stabilization

ABSTRACT

Robotic control systems and methods may include providing an end effector tool of a robotic device configured to perform a task on a work surface within a worksite coordinate frame. Unintended movement over time of the end effector tool with respect to the work surface and with respect to the worksite coordinate frame may be determined based on image data indicative of the work surface, first location data indicative of a first location of the end effector tool with respect to the worksite coordinate frame, and second location data indicative of a second location of the end effector tool with respect to the work surface. One or more control signals for the robotic device may be adjusted in order to counteract the unintended movements of the end effector tool with respect to the work surface and worksite coordinate frame.

BACKGROUND

Robotic systems may be used for applications involving materialhandling, welding, assembly, and dispensing, among others. Over time,the manner in which these robotic systems operate is becoming moreintelligent, more efficient, and more intuitive. As robotic systemsbecome increasingly prevalent in numerous aspects of modern life, theneed for robotic systems capable of working autonomously in variousenvironments becomes apparent. Therefore, a demand for such roboticsystems has helped open up a field of innovation in robot parts, sensingtechniques, as well as component design and assembly.

SUMMARY

In a worksite with robotic devices interacting with various objects, itis desirable to accurately track the location of the robotic devices andobjects. Methods and systems for monitoring and controlling roboticsystems are described herein. Example implementations may relate to acontrol system that is programmatically arranged to track the locationof various objects or robotic devices throughout a worksite. The controlsystem may also be arranged to stabilize an end effector tool mounted ona robotic arm of a robotic device. For example, by tracking the locationof the end effector tool within the worksite and with respect to a worksurface, the end effector tool may be manipulated to counteract detectedundesired movements.

In one aspect, a method is described. The method may include providingan end effector tool mounted to a moveable component of a roboticdevice. The end effector tool may be configured to perform a task on awork surface within a worksite coordinate frame. The method may furtherinclude providing first location data indicating a first location of theend effector tool with respect to the work surface, providing secondlocation data indicating a second location of the end effector toolwithin the worksite coordinate frame, and providing third location dataindicating a third location of the end effector tool within the worksitecoordinate frame. The second location data may be provided at a lowerrefresh rate than the first location data and may have a higher spatialresolution than the first location data. The third location data may beprovided at a lower refresh rate than the second location data and mayhave a higher spatial resolution than the second location data. Themethod may further include tracking the location of the end effectortool based on the first, second, and third location data, and, based onthe tracked location of the tool, instructing the robotic device tomanipulate the end effector tool to perform a task on the work surface.

In another respect, a system is disclosed. The system may include an endeffector tool mounted to a moveable component of a robotic device. Theend effector tool may be configured to perform a task on a work surfacewithin a worksite coordinate frame. The system may further include afirst location measurement system configured to provide first locationdata indicating a first location of the end effector tool with respectto the work surface, a second location measurement system configured toprovide second location data indicating a second location of the endeffector tool within the worksite coordinate frame, and a third locationmeasurement system configured to provide third location data indicatinga third location of the end effector tool within the worksite coordinateframe. The second location data may be provided at a lower refresh ratethan the first location data and may have a higher spatial resolutionthan the first location data. The third location data may be provided ata lower refresh rate than the second location data and may have a higherspatial resolution than the second location data. The system may furtherinclude a control system configured to track the location of the endeffector tool based on the first, second, and third location data, and,based on the tracked location of the tool, instruct the robotic deviceto manipulate the end effector tool to perform a task on the worksurface.

In a further respect, another method is provided. The method may includeproviding an end effector tool mounted to a moveable component of arobotic device. The end effector tool may be configured to perform atask on a work surface within a worksite coordinate frame. The methodmay further include receiving image data indicative of the work surfaceand determining, based on the image data, first movement informationindicating unintended movement over time of the end effector tool withrespect to the work surface. The method may further include receivingfirst location data indicating a first location of the end effector toolwith respect to the worksite coordinate frame and determining, based onthe first location data, second movement information indicatingunintended movement over time of the end effector tool with respect tothe worksite coordinate frame. The method may further include receivingsecond location data indicating a second location of the end effectortool with respect to the work surface and determining, based on thesecond location data, third movement information indicating unintendedmovement over time of the end effector tool with respect to the worksurface. The method may further include determining, based on the first,second, and third movement information, one or more adjustments to oneor more control signals for the robotic device in order to counteractthe unintended movements of the end effector tool with respect to thework surface and worksite coordinate frame. The method may furtherinclude communicating instructions to the robotic device to operate inaccordance with the one or more adjustments.

In yet a further aspect, another system is disclosed. The system mayinclude an end effector tool mounted to a moveable component of arobotic device. The end effector tool may be configured to perform atask on a work surface within a worksite coordinate frame. The systemmay further include a control system configured to receive image dataindicative of the work surface and determine, based on the image data,first movement information indicating unintended movement over time ofthe end effector tool with respect to the work surface. The controlsystem may be further configured to receive first location dataindicating a first location of the end effector tool with respect to theworksite coordinate frame and determine, based on the first locationdata, second movement information indicating unintended movement overtime of the end effector tool with respect to the worksite coordinateframe. The control system may be further configured to receive secondlocation data indicating a second location of the end effector tool withrespect to the work surface and determine, based on the second locationdata, third movement information indicating unintended movement overtime of the end effector tool with respect to the work surface. Thecontrol system may be further configured to determine, based on thefirst, second, and third movement information, one or more adjustmentsto one or more control signals for the robotic device in order tocounteract the unintended movements of the end effector tool withrespect to the work surface and worksite coordinate frame. The controlsystem may be further configured to communicate instructions to therobotic device to operate in accordance with the one or moreadjustments.

These as well as other aspects, advantages, and alternatives, willbecome apparent to those of ordinary skill in the art by reading thefollowing detailed description, with reference where appropriate to theaccompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 depicts a worksite coordinate frame according to an exampleembodiment.

FIG. 2 is a simplified block diagram depicting components of a roboticdevice control system, according to an example embodiment.

FIG. 3 is a simplified block diagram depicting components of a roboticdevice, according to an example embodiment.

FIG. 4 depicts a flowchart of a method according to an exampleembodiment.

FIG. 5 depicts a global location measurement system according to anexample embodiment.

FIG. 6 depicts a local location measurement system according to anexample embodiment.

FIG. 7 depicts an illustration of an optical flow field according to anexample embodiment.

FIG. 8 depicts a flowchart of a method according to an exampleembodiment.

DETAILED DESCRIPTION

Exemplary methods and systems are described herein. It should beunderstood that the word “exemplary” is used herein to mean “serving asan example, instance, or illustration.” Any embodiment or featuredescribed herein as “exemplary” or “illustrative” is not necessarily tobe construed as preferred or advantageous over other embodiments orfeatures. In the figures, similar symbols typically identify similarcomponents, unless context dictates otherwise. The example embodimentsdescribed herein are not meant to be limiting. It will be readilyunderstood that the aspects of the present disclosure, as generallydescribed herein, and illustrated in the figures, can be arranged,substituted, combined, separated, and designed in a wide variety ofdifferent configurations, all of which are contemplated herein.

I. OVERVIEW

A worksite may include a number of robotic devices and various objects.The robotic devices may be configured to carry out various tasks bymoving about the worksite and interacting with the objects. In order tomost effectively control the robotic devices, it is desirable to trackthe positions of the robotic devices and various objects within theworksite and with respect to each other.

Illustrative implementations relate to a control system that trackslocations of target objects throughout a worksite. In particular, thecontrol system may operate in a worksite, such as a construction sitefor instance, and in doing so, the control system may have sensors thatprovide position information for at least one target object within theworksite. These target objects may include robotic devices, componentsor linkages of robotic devices, tools, and/or construction materials,among other objects. In some embodiments, a target object may be, but isnot limited to, an end effector tool mounted to a robotic arm of aworker robot. The worker robot may be configured to manipulate the endeffector tool to perform a task on a work surface, such as drilling,milling, screwing, welding, gripping, etc. As such, the control systemmay dynamically track the location of the end effector tool within theworksite and with respect to the work surface to ensure that the endeffector tool is performing a desired task.

The control system may include a number of global sensors for trackingthe location of the end effector tool within a coordinate frame of theworksite. The global sensors may provide location data with high spatialaccuracy, but at a low rate. The global sensors may include a lasertracker system, a motion capture system, a visual system, and/or atime-of-flight system for determining positions of markers in theworksite. The markers may be placed at various locations throughout theworksite to form the coordinate frame, and they may be attached to thetarget objects as well, such as the end effector tool. The globalsensors may be located in fixed locations throughout the worksite, orthey may be attached to moveable devices. For example, the globalsensors may be attached to mobile observer robots capable of movingabout the worksite.

The control system may further include a number of local sensors fortracking the location of the end effector tool with respect to the worksurface. The local sensors may provide location data with a spatialaccuracy lower than that of the global sensors, but at a higher rate.The local sensors may include a camera mounted on the robotic arm nearthe end effector tool for capturing video of the work surface. Thecaptured video may be analyzed using optical flow processes to determinea position of the end effector tool with respect to the work surface.Since optical flow processes only provide two-dimensional positionalinformation, multiple optical flow systems may be applied in differentdirections, or the local sensors may further include a distance sensor,such as a laser distance sensor, for determining a third dimension ofthe position of the end effector tool with respect to the work surface.The local sensors may further include an inertial measurement unit (IMU)mounted to or near the end effector tool. By integrating data from theIMU over time, the position of the end effector tool with respect to thework surface may be determined.

While the local sensors may facilitate determining the position of theend effector tool at a higher rate than the global sensors, the localsensors may have a lower spatial accuracy than the higher sensorsbecause the position of the end effector tool is determined relative tothe work surface. Consequently, the local sensors are subject toinaccuracies due to error accumulation and drift. Further, the localsensors may provide data with a lower spatial resolution than data fromthe global sensors. By determining the location of the end effector toolbased on a combination of data provided by local and global sensors, theinaccuracies of the local sensor data may be reduced or removed.

Further, in some embodiments, the control system may stabilize the endeffector tool based on data from the local and global sensors. Bycombining the data from the local and global sensors to determine theposition of the end effector tool, the control system may determine thatthe end effector tool was moved in an undesired manner. For example, anundesired movement may be movement caused by vibrations, the roboticdevice sinking into the ground, or any other unintentional movement ofthe end effector tool. To stabilize the end effector tool, the controlsystem may provide instructions to the robotic device to manipulate theend effector tool in a manner to counteract the undesired movement.

It should be understood that the above examples are provided forillustrative purposes, and should not be construed as limiting. As such,the method may additionally or alternatively include other features orinclude fewer features, without departing from the scope of theinvention.

II. EXAMPLE SYSTEMS AND METHODS

A. Example Worksite Coordinate Frame

FIG. 1 depicts a worksite coordinate frame 100, according to an exampleembodiment. The worksite coordinate frame 100 may define a portion of anenvironment in which objects, machines, and perhaps humans may belocated. The worksite coordinate frame 100 may take on athree-dimensional form and may be used for various purposes. Forinstance, the worksite coordinate frame 100 may be defined for aconstruction site where the construction of a building or anotherproject is being or is about to be carried out.

While various aspects of the disclosure are discussed below in thecontext of a construction site, example implementations are not limitedto construction sites and may extend to a variety of other worksitecoordinate frames, such as retail spaces, manufacturing facilities,distribution facilities, office spaces, shopping centers, festivalgrounds, and/or airports, among other examples. Additionally, while oneworksite coordinate frame 100 is shown in FIG. 1, exampleimplementations may be carried out in the context of a number ofworksite coordinate frames.

As depicted in FIG. 1, the worksite coordinate frame 100 includes anumber of observer robots 110, a mover robot 120, a worker robot 130,and construction materials 140.

The construction materials 140 may be any materials or tools located ina construction site. For example, in FIG. 1, the construction materials140 are depicted as a palette of bricks. Other examples of constructionmaterials 140 may include, but are not limited to: drywall, screws,nails, mortar, cement, water, soil, gravel, glass, metal framing, metalsheets, bonding materials, landscaping materials, etc.

The worker robot 130 may be a robotic device configured to perform atask within the worksite coordinate frame 100. In the illustratedscenario, the worker robot 130 includes an end effector tool mounted toa robotic arm. The end effector tool may be configured to perform a taskon a work surface, such as drilling, milling, welding, nailing,riveting, sanding, spraying, gripping, or any other task typicallyperformed during construction of a building. Further, the robotic arm ofthe worker robot 130 may include a mount to which different types of endeffectors can be attached. As such, different end effectors may beswapped out such that the worker robot 130 can perform different typesof tasks. Further, the worker robot 130 may be capable of movingthroughout the worksite. For example, as depicted in FIG. 1, the workerrobot 130 may include wheels. However, other configurations forproviding mobility are possible as well (e.g., biped devices, quadrupeddevices, treads, tracks, etc.).

The mover robot 120 may be a robotic device configured to move itemsthroughout the worksite coordinate frame 100. For example, the moverrobot 120 may be configured to move the construction materials 140 fromone location within the worksite coordinate frame 100 to another.Further, the mover robot 120 may be capable of moving throughout theworksite. For example, as depicted in FIG. 1, the mover robot 120 mayinclude wheels. However, other configurations for providing mobility arepossible as well (e.g., biped devices, quadruped devices, treads,tracks, etc.).

The observer robots 110 may be configured to move about the worksitecoordinate frame 100. For example, as depicted in FIG. 1, the observerrobots 110 may be biped robotic devices capable of walking throughoutthe worksite coordinate frame 100. Alternatively, the observer robots110 may include wheels, tracks, treads, or any other component thatprovides mobility.

The observer robots 110 may include sensors that facilitate tracking thelocation of target objects within the worksite coordinate frame 100. Forexample, the observer robots 110 may include laser tracker systems totrack the location of the mover robot 120, worker robot 130, andconstruction materials 140 within the worksite coordinate frame 100.Laser trackers may be used to measure the location of objects bydirecting a laser beam at a retroreflective marker attached to themeasured object. Light reflects off the retroreflective marker andreturns to the laser tracker. The laser tracker may determine thelocation of the marker based on characteristics of the reflected light.Laser trackers may measure location with very high accuracy, but themeasurement data may be provided at a low refresh rate. For example, alaser tracker may measure the location of an object several meters awaywith a resolution of a few hundredths of a millimeter, but themeasurement may be provided at a refresh rate of the order of a fewhertz.

In FIG. 1, the worksite coordinate frame 100 includes a number of pylonmarkers 104. The pylon markers 104 may be located at the boundaries ofthe worksite coordinate frame 100 and/or throughout the worksitecoordinate frame 100 and may be used to establish a three-dimensionalcoordinate system 102 within the worksite coordinate frame 100. Forexample, the three-dimensional coordinate system 102 may be a Cartesiancoordinate system with an x-axis, a y-axis, and a z-axis. One of thepylon markers 104 may be designated as the origin of the coordinatesystem 102, and the remaining pylon markers 104 may be spaced a knowndistance from the origin. As a result, each of the pylon markers 104 maybe associated with a known (x, y, z) coordinate within the coordinatesystem 102, where the x-, y-, and z-coordinates correspond to a distancefrom the origin pylon marker along the x-, y-, and z-axes respectively.

The pylon markers 104 do not necessarily need to be located at theboundaries of the worksite coordinate frame 100, but may alternativelyor additionally be arranged at various known locations throughout theworksite coordinate frame 100. For example, in some embodiments, thepylon markers 104 may be arranged in a two-dimensional orthree-dimensional grid throughout the worksite. However, otherconfigurations are possible as well, and the pylon markers 104 may bearranged in any manner of known locations in the worksite coordinateframe 100.

The pylon markers 104 may be retroreflective such that the laser trackerof an observer robot 110 may measure the location of the pylon markers104 with respect to the observer robot 110. By determining the locationof a pylon marker with known coordinates from an observer robot 110, thecoordinates of the observer robot 110 may be derived. As an observerrobot 110 moves about the worksite coordinate frame 100, it mayoccasionally provide a line of sight between its laser tracker and apylon marker 104. This provides updated coordinates for the location ofthe observer robot 110 as it moves about the worksite coordinate frame100.

In addition to the pylon markers 104, the worksite coordinate frame 100may include a number of additional markers 112. The markers 112 may beattached to various target objects throughout the worksite coordinateframe 100. For example, as depicted in FIG. 1, respective markers 112may be attached to the observer robots 110, the mover robot 120, theworker robot 130, and/or the construction materials 140. The location ofthe markers 112 may be measured to provide coordinates within thecoordinate system 102 associated with the mover robot 120, worker robot130, and/or construction materials 140. For example, as depicted in FIG.1, a laser tracker in an observer robot 110 may direct a laser toward aretroreflective marker 112 attached to the mover robot 120 along a lineof sight 114. By analyzing the reflected light, the laser tracker maydetermine the position of the retroreflective marker 112 with respect tothe observer robot 110. Since the coordinates of the observer robot 110within the coordinate system 102 are known, the position of theretroreflective marker 112 with respect to the observer robot 110 may beused to determine coordinates of the retroreflective marker 112 withinthe coordinate system 102. By determining coordinates of the markers112, the locations of the mover robot 120, worker robot 130, andconstruction materials 140 within the worksite coordinate frame 100 maybe tracked.

Similarly, an observer robot 110 may provide a line of sight 116 betweenitself and another observer robot. In this manner, the observer robots110 may facilitate determining the location of other observer robots110, and the location of the observer robots 110 within the worksitecoordinate frame 100 may be tracked.

In some embodiments, determining the location of a target object in theworksite coordinate frame 100 may involve more than simply determiningthe location of a single point within a three-dimensional coordinatesystem. For instance, in some embodiments, the locations of a set ofpoints may be determined to define a volume of the target object. Forexample, referring to FIG. 1, a three-dimensional space representing thevolume of the construction materials 140 may be determined. The observerrobots 110 may determine the locations of a set of markers 112 attachedto the construction materials 140. The markers 112 may be attached tothe boundaries or edges of the construction materials 140, or in someembodiments, the markers 112 may be arranged on the constructionmaterials 140 in any known manner. By determining the location of theset of markers 112 arranged on the construction materials 140, thelocation of a three-dimensional volume may be determined indicating ashape of the construction materials 140 within the worksite coordinateframe 100. Similarly, by determining the locations of sets of markers112 arranged on various target objects, three-dimensional volumes andtheir positions within the worksite coordinate frame 100 may bedetermined for the mover robot 120, the worker robot 130, the observerrobots 110, or any other object in the worksite coordinate frame 100.

In some embodiments, determining the location of a target object in theworksite coordinate frame 100 may include determining a pose of thetarget object relative to the worksite coordinate frame 100. The pose ofthe target object may include a combination of the position andorientation of the object. Various processes may be used to determinethe pose of a target object, including analytic or geometric methods,genetic algorithm methods, and/or learning-based methods, among others.

In some embodiments, where the target object is a robotic device, thepose of the robot may be determined based on its operational state. Therobotic device may have various operational states that result indifferent poses. A control system may determine the operational state ofthe robotic device. Given that the volume and/or shape of the roboticdevice is already known or has otherwise been determined, the controlsystem may determine the pose of the robotic device based on thedetermined operational state.

For example, referring to FIG. 1, the worker robot 130 includes arobotic arm that may be positioned in any number of poses based on itsoperational state. One operational state may include configuring therobotic arm to perform a task on a work surface. However, an operationalstate may include any configuration of a robotic device in the worksitecoordinate frame 100 that results in an associated pose. The volume andshape of the various components of the robotic arm are known or haveotherwise been determined. Thus, by determining the pose (e.g., theposition and orientation) of the robotic arm based on the operationalstate of the worker robot 130, a three-dimensional volume representingthe space occupied by the robotic arm within the worksite coordinateframe 100 may be determined.

B. Example Robotic Control Systems

FIG. 2 is a simplified block diagram depicting components of a roboticdevice control system 200, according to an example embodiment. Therobotic device control system 200 includes a robotic device 210configured to manipulate an end effector tool 220. The end effector tool220 may be configured to perform a task on a work surface and may bemounted to a moveable component, such as a robotic arm, of the roboticdevice 210. The robotic device 210 may be located within a worksitecoordinate frame, such as the worksite coordinate frame 100 depicted inFIG. 1, and the robotic device 210 may, for example, be a worker robot,such as the worker robot 130 depicted in FIG. 1.

The robotic device control system 200 may further include localsensor(s) 230 and global sensor(s) 240 configured to determine thelocation of various objects in the coordinate frame of a worksite, suchas the end effector tool 220, for example. The local sensor(s) 230 maybe arranged on or within the robotic device 210 and may be configured tomeasure the location of the end effector tool 220 with respect to thework surface. The global sensor(s) 240, on the other hand, may bearranged within the worksite coordinate frame separate from the roboticdevice 210 and may be configured to measure the location of the endeffector tool 220 with respect to the worksite coordinate frame.

The local sensor(s) 230 may provide location measurements of the endeffector tool 220 at a higher refresh rate than the global sensor(s)240, but may not be as accurate due to lower sensor resolution and/orerror buildup. For example, the global sensor(s) 240 may include a lasertracker system with very high resolution (e.g., hundredths of amillimeter) and a low refresh rate (e.g., a few hertz), such as thelaser tracker system included in the observer robots 110 in FIG. 1. Andthe local sensor(s) 230 may include a high speed camera for providingoptical flow data or an inertial measurement unit (IMU) with a highrefresh rate (e.g., 250 Hz for the camera and 1 kHz for the IMU) butlower accuracy due to drift and accumulation of error over time.

The global sensor(s) 240 are not limited to laser tracker systems, butmay include any sensor capable of measuring the location of the endeffector tool 220 with respect to the worksite coordinate frame, such asmotion capture sensors, light detection and ranging (LIDAR) sensors,point cloud sensors, ultrasonic range sensors, Global Positioning System(GPS) receivers, sonar, optical sensors, Radio Frequency identification(RFID) systems, Near Field Communication (NFC) chips, wireless sensors,radio sensors, radars, cameras (e.g., color cameras, grayscale cameras,and/or infrared cameras), and/or range sensors (e.g., ultrasonic and/orinfrared), among others.

Similarly, the local sensor(s) 230 are not limited to high speed camerasor IMUs, but may include any sensor capable of measuring the location ofthe end effector tool 220 with respect to the work surface, such asforce sensors, proximity sensors, motion sensors (e.g., gyroscopes,and/or accelerometers), load sensors, position sensors, thermal imagingsensors, depth sensors (e.g., RGB-D, laser, structured-light, and/or atime-of-flight camera), ultrasonic range sensors, infrared sensors,optical sensors, Radio Frequency identification (RFID) systems, NearField Communication (NFC) chips, wireless sensors, light sensors, touchsensors (e.g., capacitive sensors), cameras (e.g., color cameras,grayscale cameras, and/or infrared cameras), and/or range sensors (e.g.,ultrasonic and/or infrared), among others.

Additionally, the local sensor(s) 230 and global sensor(s) 240 may bepositioned within or in the vicinity of the worksite coordinate frame,among other possible locations. For example, the local sensor(s) 230 maybe attached to the robotic device 210, and the global sensor(s) 240 maybe attached to a different robotic device capable of moving throughoutthe worksite coordinate frame, such as the observer robots 110 depictedin FIG. 1. In some embodiments, the global sensor(s) 240 may be arrangedin fixed locations throughout the worksite coordinate frame, forexample, as a dedicated sensing installation. Further, an exampleimplementation may also use sensors incorporated within existingdevices, such as mobile phones, laptops, and/or tablets. These devicesmay be in possession of people located in the worksite coordinate frame,such as construction workers in a construction site.

FIG. 2 also depicts a controller 250 that may receive data from thelocal sensor(s) 230 and global sensor(s) 240. In particular, the localsensor(s) 230 and global sensor(s) 240 may provide sensor data to thecontroller 250 through a communication unit 260. The communication unit260 may include wired links and/or wireless links (e.g., using variouswireless transmitters and receivers). A wired link may include, forexample, a parallel bus or a serial bus such as a Universal Serial Bus(USB). A wireless link may include, for example, Bluetooth, IEEE802.11(IEEE 802.11 may refer to IEEE 802.11-2007, IEEE 802.11n-2009, orany other IEEE 802.11 revision), Cellular (such as GSM, GPRS, CDMA,UMTS, EV-DO, WiMAX, HSPDA, or LTE), or Zigbee, among otherpossibilities. Furthermore, multiple wired and/or wireless protocols maybe used, such as “3G” or “4G” data connectivity using a cellularcommunication protocol (e.g., CDMA, GSM, or WiMAX, as well as for “WiFi”connectivity using 802.11).

In other examples, the robotic control system 200 may include accesspoints through which the local sensor(s) 230 and global sensor(s) 240and/or controller 250 may communicate with a cloud server. Access pointsmay take various forms such as the form of a wireless access point (WAP)or wireless router. Further, if a connection is made using a cellularair-interface protocol, such as a CDMA or GSM protocol, an access pointmay be a base station in a cellular network that provides Internetconnectivity via the cellular network. Other examples are also possible.

The controller 250 is shown to include one or more processor(s) 252,data storage 254, program instructions 256, an input/output unit 258,and a power source 262. Note that the controller 250 is shown forillustration purposes only, as the controller 250 may include additionalcomponents and/or have one or more components removed without departingfrom the scope of the disclosure. Further, note that the variouscomponents of the controller 250 may be arranged and connected in anymanner. The controller 250 may be incorporated in whole or in part intothe robotic device 210 or may take the form of a desktop computer, alaptop, a tablet, a wearable computing device, and/or a mobile phone,among other possibilities.

Each processor, from the one or more processor(s) 252, may be ageneral-purpose processor or a special purpose processor (e.g., digitalsignal processors, application specific integrated circuits, etc.). Theprocessor(s) 252 can be configured to execute computer-readable programinstructions 256 that are stored in the data storage 254 and areexecutable to provide the functionality of the controller 250 describedherein. For instance, the program instructions 256 may be executable toprovide for processing of sensor data received from the local sensor(s)230 and global sensor(s) 240.

The data storage 254 may include or take the form of one or morecomputer-readable storage media that can be read or accessed by theprocessor(s) 252. The one or more computer-readable storage media caninclude volatile and/or non-volatile storage components, such asoptical, magnetic, organic or other memory or disc storage, which can beintegrated in whole or in part with the processor(s) 252. In someembodiments, the data storage 254 can be implemented using a singlephysical device (e.g., one optical, magnetic, organic or other memory ordisc storage unit), while in other embodiments, the data storage 254 canbe implemented using two or more physical devices. Further, in additionto the computer-readable program instructions 256, the data storage 254may include additional data such as diagnostic data, among otherpossibilities. Further, the controller 250 may also include a powersource 262 configured to supply power to various components of thecontroller 250. Any type of power source may be used, such as directcurrent from a battery or alternating current from mains electricity.

FIG. 2 further depicts the controller 250 including an input/output unit258. The input/output unit 258 may output information to a user througha display. The display may take on any form and may be arranged toproject images and/or graphics to a user of the controller 250. In anexample arrangement, a projector within the input/output unit 258 may beconfigured to project various projections of images and/or graphics ontoa surface of the display. The display may include: an opaque or atransparent (or semi-transparent) matrix display, such as anelectroluminescent display or a liquid crystal display, one or morewaveguides for delivering an image to the user's eyes, or other opticalelements capable of delivering an image to the user. A correspondingdisplay driver may be disposed within the controller 250 for drivingsuch a matrix display. Other arrangements may also be possible for thedisplay. As such, the display may show a graphical user interface (GUI)that may provide an application through which the user may interact withthe systems disclosed herein.

Additionally, the input/output unit 258 may receive user-input (e.g.,from the user of the controller 250). In particular, the input/outputunit 258 may allow for interaction with the GUI such as for scrolling,providing text, and/or selecting various features of the application,among other possible interactions. The input/output unit 258 may take onvarious forms. In one example, the input/output unit 258 may include apointing device such as a computing mouse used for control of the GUI.However, if the input/output unit 258 includes a touch screen display,touch-input can be received (e.g., such as using a finger or a stylus)that allows for control of the GUI. In another example, the input/outputunit 258 may include a keyboard that provides for selection of numbers,characters and/or symbols to be displayed via the GUI. For instance, inthe arrangement where the input/output unit 258 includes a touch screendisplay, portions the display may show the keyboard. Thus, touch-inputon the portion of the display including the keyboard may result inuser-input such as selection of specific numbers, characters, and/orsymbols to be shown on the GUI through the display. In yet anotherexample, the input/output unit 258 may include a voice input device thatreceives audio input, such as from a user through a microphone, that isthen interpretable using one of various speech recognition techniquesinto one or more characters that may be shown through the display. Otherexamples may also be possible.

C. Example Robotic Device

FIG. 3 shows an example configuration of a robotic device 300, whichcould take the form an observer robot 110, a mover robot 120, or aworker robot 130, among other possibilities. Generally, a robotic device300 may be any device that has a computing ability and interacts withits surroundings with an actuation capability and/or with ability toemit/generate physical phenomena such as light and/or sound, amongothers. For instance, the robotic device 300 may be a humanoid robot, arobotic arm, or a quadruped robot, among others. A robotic device mayalso be any device that is generally understood to those of ordinaryskill in the art as being a “robotic.” The robotic device 300 may alsobe referred to as a robotic device, a robotic manipulator, a robotclient, or a robot, among others.

The robotic device 300 is shown to include processor(s) 302, datastorage 304, program instructions 306, controller 308, sensor(s) 310,power source(s) 312, actuator(s) 314, and movable component(s) 316. Notethat the robotic device 300 is shown for illustration purposes only androbotic device 300 may include additional components and/or have one ormore components removed without departing from the scope of thedisclosure. Further, note that the various components of robotic device300 may be arranged and connected in any manner.

Moreover, the above description of processor(s) 252, data storage 254,program instructions 256, sensors (e.g., local sensor(s) 230 and globalsensor(s) 240), and/or power source 262, may apply to any discussionbelow relating to the respective component being used in another systemor arrangements. For instance, as noted, FIG. 3 (among other possiblefigures) illustrates processors, data storage, program instructions,sensors, and/or power as being incorporated in another arrangement.These components at issue may thus take on the same or similarcharacteristics (and/or form) as the respective components discussedabove in association with FIG. 2. However, the components at issue couldalso take on other characteristics (and/or form) without departing fromthe scope of the disclosure.

As noted, the robotic device 300 may include a controller 308 (e.g.,taking the form of a microcontroller). The controller 308 may includeprocessing unit and data storage, and may be arranged to manage or carryout various operations (e.g., individually or in collaboration withprocessor(s) 302). Thus, this controller 308 could take on the same orsimilar characteristics (and/or form) as the above-mentioned controller250, but could take on other characteristics (and/or form) as well. Soin some implementations, the controller 250 may be incorporated as partthe robotic device 300 and thus controller 250 may itself be controller308. In other implementation, controller 308 may be included as part ofthe robotic device 300 and controller 250 may be separate from therobotic device 400. Regardless of the implementations, these controllersmay take various forms. For instance, a controller may take the form ofa chip set, a server system, a digital signal processor, a programmablelogic controller, and/or a sampled-data system, among otherpossibilities. Moreover, a controller could also be referred to hereinas a control system, among other.

Additionally, the robotic device 300 may also include one or moreactuator(s) 314. An actuator is a mechanism that may be used tointroduce mechanical motion. In particular, an actuator may beconfigured to convert stored energy into movement of one or morecomponents. Various mechanisms may be used to power an actuator. Forinstance, actuators may be powered by chemicals, compressed air,hydraulics, or electricity, among other possibilities. With thisarrangement, actuator(s) 314 may cause movement of various movablecomponent(s) 316 of the robotic device 300. The moveable component(s)316 may include appendages/members such as robotic arms, legs, and/orhands, among others. The moveable component(s) 316 may also include amovable base, wheels, and/or end effectors, among others. Further, whena robotic device 300 includes at least one end effector, such an endeffector may be a tool (e.g., a screwdriver, drill, welding iron, orsome combination thereof) and/or a gripper, among others as discussedabove.

D. Example Multi-Resolution Localization Systems and Methods

FIG. 4 depicts a flowchart of a method 400 according to an exampleembodiment. The method 400 may be implemented to track the location ofan end effector tool of a robotic device.

Step 402 involves providing a robotic device with an end effector toolmounted to a moveable component of the robotic device. The roboticdevice may take the form of the robotic device 300 depicted in FIG. 3,or, more specifically, the robotic device may take the form of theworker robot 130 depicted in FIG. 1. The end effector tool may beconfigured to perform a task on a work surface located within a worksitecoordinate frame, such as the worksite coordinate frame 100 depicted inFIG. 1.

Step 404 involves measuring a first location of the end effector toolwith respect to the work surface. The first location may be determinedbased on location data provided by a local sensor, such as one of thelocal sensor(s) 230 depicted in FIG. 2. The first location data may beprovided at a high refresh rate (e.g., 250 Hz or higher), but may have alow spatial accuracy due to error accumulation and/or low precisionsensors. For example, the first location may be determined by performinga dead reckoning analysis on optical flow data provided by a cameramounted on the robotic device, as described in more detail below withreference to FIGS. 6 and 7.

Step 406 involves measuring a second location of the end effector toolwith respect to the worksite coordinate frame. The second location maybe determined based on location data provided by a first global sensor,such as one of the global sensor(s) 240 depicted in FIG. 2. The secondlocation data may be provided at a lower refresh rate (e.g.,approximately 100 Hz) than the first location data, but may have ahigher spatial accuracy than the first location data because it is notsubject to error accumulation (and/or for other reasons). For example,the second location may be determined by applying motion captureprocesses to image data (e.g., video) captured by cameras in theworksite environment, as described in more detail below with referenceto FIG. 5.

Step 408 involves measuring a third location of the end effector toolwith respect to the worksite coordinate frame. The third location may bedetermined based on location data provided by a second global sensor,such as one of the global sensor(s) 240 depicted in FIG. 2. The thirdlocation data may be provided at a lower refresh rate (e.g.,approximately 1 Hz) than the second location data, but may have a higherspatial accuracy than the second location data. The third location datamay have a higher spatial accuracy because it is not subject to erroraccumulation, or is subject to comparatively less error accumulation,and is generated based on sensors providing higher precision data. Forexample, a laser tracker system, which provides highly accurate locationinformation, may be implemented as the third source of location data asdescribed in more detail below with reference to FIG. 5.

Step 410 involves tracking the location of the end effector tool basedon the first, second, and third location measurements. This may involvefirst tracking the location of the end effector tool based on the firstlocation measurement. The local sensor providing the first locationmeasurement may provide the first location measurement at a high refreshrate. For example, the local sensor may be a camera providing opticalflow data at a refresh rate of approximately 250 Hz. Over time, as thelocal sensor continues to provide location data, the first locationmeasurement may become more inaccurate due to error accumulation, drift,and/or due to the precision tolerance of the local sensor.

The second location measurement may be used to correct for inaccuraciesin the first location measurement. The first global sensor providing thesecond location measurement may provide second location measurement dataat a lower refresh rate than the local sensor, but the second locationdata may have a higher spatial accuracy than the first location data.For example, the first global sensor may be a motion capture system. Inexample embodiments, “motion capture” systems utilize data captured fromimage sensors to triangulate the three-dimensional position of a target.The target may be indicated by markers, such as the markers 112 depictedin FIG. 1. A motion capture system may have a lower refresh rate (e.g.,approximately 100 Hz) than the local sensor, but its spatial accuracymay be higher because it does not suffer from error accumulation.Therefore, the first global sensor may provide feedback data for acalibrating process that accounts and/or adjusts for errors in thelocally measured first location data arising due to accumulatedmeasurement error.

The third location measurement may of even higher precision, as comparedto the second location measurement, but may be obtained less frequentlydue to a lower refresh rate. As such, the third location data may beutilized to correct for lower tolerances in the first global sensormeasurements and/or accumulation of errors in locations data from localsensors. The second global sensor providing the third locationmeasurement may provide third location data at a lower refresh rate thanthe first global sensor, but the third location data may have a higherspatial accuracy than the second location data. For example, the secondglobal sensor may be a laser tracker system configured to providelocation data of an object several meters away with a resolution of afew hundredths of a millimeter, but the location data may be provided ata refresh rate of the order of a few hertz or lower. Therefore, thesecond global sensor may serve as a calibrating feedback to account forthe lower resolution of the first global sensor.

By determining the location of the end effector tool based on thecombination of the first, second, and third location measurement data,method 400 can update location information for the end effector tool ata high rate (e.g., 250 Hz or higher) with a higher accuracy than wouldbe possible using any single source of location information.

FIG. 5 depicts a global location measurement system 500 according to anexample embodiment. The global location measurement system may beconfigured to measure the location of an end effector tool 512 withrespect to a worksite coordinate frame, such as the worksite coordinateframe 100 depicted in FIG. 1. The end effector tool 512 may beconfigured to perform a task on a work surface 510 and may be mounted ona moveable component, such as a robotic arm 514, of a robotic device510. The robotic device 510 may be the robotic device 300 depicted inFIG. 3, or more specifically, the robotic device may be the worker robot130 depicted in FIG. 1.

The global location measurement system 500 may include one or moremarkers 112. The markers 112 may be mounted at various locations on therobotic device 510, such as on corners of a base of the robotic device510, on joints of the robotic arm 514, on the end effector tool 512,and/or on some other location of the robotic device 510.

The global location measurement system 500 may further include a globalsensor (not shown), such as the global sensor(s) 240 depicted in FIG. 2.In some embodiments, the markers 112 may be retroreflective markers, andthe global location measurement system 500 may include a laser trackerfor tracking the retroreflective markers 112. The laser tracker may bemounted on a moveable robot, such as the observer robot 110 depicted inFIG. 1. The observer robot 110 may direct a laser at a retroreflectivepylon marker 104 with predetermined three-dimensional coordinates withinthe worksite coordinate frame 100. Based on the reflected light, thelaser tracker may determine location data indicating the position of thepylon marker 104 with respect to the laser tracker. The location datamay be communicated to a controller, such as the controller 250 depictedin FIG. 2. Based on the location data of the pylon marker 104, thecontroller 250 may determine three-dimensional coordinates of the lasertracker on the observer robot 110 within the worksite coordinate frame100.

The observer robot 110 may further direct a laser at one of theretroreflective markers 112, such as a marker 112 mounted on the endeffector tool 512. Based on the reflected light, the laser tracker maydetermine location data indicating the position of the marker 112 withrespect to the laser tracker. The location data may be communicated tothe controller 250, and, based on the location data, the controller 250may determine three-dimensional coordinates of the marker 112 mounted onthe end effector tool 512 within the worksite coordinate frame 100. Themarker 112 may be affixed to the end effector tool 512 such that themarker 112 maintains a constant known position relative to the endeffector tool 512. By determining the coordinates of the marker 112, thecontroller 250 may determine coordinates of the end effector tool 512based on the known relative positions of the marker 112 and the endeffector tool 512. Location data indicating the coordinates of the endeffector tool 512 may be stored in the data storage 254 of thecontroller 250.

In some embodiments, there may not be a line of sight between theobserver robot 110 and the marker 112 mounted on the end effector tool512. For example, the end effector tool 512 may be performing a task ona work surface 520, and the robotic device 510 may manipulate therobotic arm 514 in a manner that moves the end effector tool 512 out ofthe line of sight of the observer robot 110. Without a line of sight,the laser tracker on the observer robot 110 may not measure the positionof the marker 112 mounted on the end effector tool 512. To account forthis, the robotic device 510 may be configured to occasionallymanipulate the end effector tool 512 to restore a line of sight betweenthe observer robot 110 and the marker 112 on the end effector tool 512.

For example, the controller 250 may store location data indicating themost recent measured position of the marker 112 on the end effector tool512. When the line of sight between the observer robot 110 and themarker 112 on the end effector tool 512 is broken, the controller 250may occasionally (e.g., once every 30 seconds) communicate instructionsto the robotic device 510 to manipulate the end effector tool 512 in amanner that returns the marker 112 on the end effector tool 512 to themost recent measured position. Once the line of sight is restored, thelaser tracker may measure the location of the marker 112 on the endeffector tool 512, the observer bot 110 may communicate new locationdata indicating the position of the marker 112 on the end effector tool512 to the controller 250, and the controller 250 may store the newlocation data in the data storage 254.

In some embodiments, the controller 250 may communicate instructions tothe robotic device 510 to manipulate the end effector tool 512 torestore line of sight more frequently or less frequently than once every30 seconds. The rate may be based on the accuracy of the local sensors230. For example, if the local sensors 230 have a very low accuracy orlarge error due to error accumulation, then line of sight may berestored more frequently than once every 30 seconds. On the other hand,if the local sensors 230 have somewhat high accuracy or low erroraccumulation, then line of sight may be restored less frequently thanonce every 30 seconds.

In some embodiments, the controller 250 may communicate instructions toa worker robot in order to manipulate the worker robot to restore lineof sight between an observer robot and the worker robot. In someembodiments, the controller 250 may communicate instructions to anobserver robot in order to manipulate the observer robot to restore lineof sight between the observer robot and a worker robot. Still, in someembodiments, the controller 250 may communicate instructions to both anobserver robot and a worker robot to manipulate both the observer robotand the worker robot to restore line of sight between the observer robotand the worker robot.

In some embodiments, the global location measurement system 500 mayinclude a motion capture system. The motion capture system may includecameras mounted on the observer robots 110 and may be configured tomeasure the position of the markers 112. For example, the globallocation measurement system 500 may include one or more referencemarkers (not shown) that may be mounted to fixed locations throughoutthe worksite coordinate frame 100. The motion capture system may measurethe movement of the marker 112 on the end effector tool 512 with respectto the one or more reference markers. Data indicating the movement ofthe marker 112 with respect to the one or more reference markers may becommunicated to the controller 250. Since the one or more referencemarkers may be mounted in fixed locations, they may have predeterminedcoordinates within the worksite coordinate frame 100. Based on themovement data indicating movement of the marker 112 on the end effectortool 512 and the predetermined coordinates of the one or more referencemarkers, the controller 250 may determine the coordinates of the marker112 on the end effector tool within the worksite coordinate frame 100.Location data indicating the coordinates of the end effector tool 512may be stored in the data storage 254 of the controller 250.

In some embodiments, the global location measurement system 500 mayinclude a LIDAR sensor. The LIDAR sensor may be mounted on the observerrobot 110 and may be configured to measure the position of the endeffector tool 512 with respect to the worksite coordinate frame 100. Forexample, the LIDAR sensor may direct light toward the end effector tooland toward some fixed object with a known location and known coordinateswithin the worksite coordinate frame 100. By analyzing the reflectedlight, the LIDAR sensor may provide data indicating the position of theend effector tool 512 with respect to the fixed object. Based on thedata indicating the position of the end effector tool 512 with respectto the fixed object, the controller 250 may determine coordinates of theend effector tool 512 within the worksite coordinate frame 100. Locationdata indicating the coordinates of the end effector tool 512 may bestored in the data storage 254 of the controller 250.

In some embodiments, the configuration of a global sensor 240 may bereversed. That is, the global sensor 240 may be mounted on a roboticdevice, such as on an end effector tool or some other component orlinkage. In this configuration, the global sensor 240 may localizeitself by scanning the environment around it. In some embodiments, theglobal sensor 240 may be localized by comparing the scan results to aknown environmental model. In some embodiments, the global sensor 240may be localized by performing simultaneous localization and mapping(SLAM), where a map of the environment is constructed whilesimultaneously keeping track of the global sensor's location within theenvironment.

FIG. 6 depicts a local location measurement system 600 according to anexample embodiment. The local location measurement system 600 may beconfigured to measure the location of an end effector tool 512 withrespect to a work surface 520, and the local location measurement system600 may provide location data indicating the location of the endeffector tool 512 at a higher refresh rate than the global locationmeasurement system 500. The end effector tool 512 may be mounted to arobotic arm 514 of a robotic device. The robotic device may be therobotic device 300 depicted in FIG. 3, or more specifically, the roboticdevice may be the worker robot 130 depicted in FIG. 1. The locallocation measurement system may include a camera 610, a distance sensor612, and a motion sensor 620.

The camera 610 may be arranged on the robotic arm 514 such that the worksurface 520 is within the field of view of the camera 610 when the endeffector tool 512 is performing a task on the work surface 520. Forexample, as depicted in FIG. 6, the camera 610 may be positionedadjacent to the end effector tool 512 with the lens directed toward theworking end of the end effector tool 512. In some embodiments, thecamera 610 may be located farther away from the end effector tool 512.For example, when the end effector tool 512 performing a task on thework surface causes poor visibility (e.g., due to dust from drilling,bright light from welding, or any other cause of poor visibility), itmay be desirable to position the camera 610 on the robotic arm 514farther away from the end effector tool 512, but close enough to the endeffector tool 512 that the work surface 520 remains in the field of viewof the camera 610.

The camera 610 may be configured to capture image data of the worksurface 520. For example, the camera 610 may be a video cameraconfigured to capture video of the work surface 520. The captured imagedata may be communicated to the controller 250 by a communication unitof the robotic device. The controller 250 may include programinstructions 256 for performing optical flow analysis to determine theoptical flow of the captured image data. As is understood by thoseskilled in the art, the “optical flow” of a video is the apparent motionof objects in the video that results from the relative movement betweenthe object and the video camera.

Further, the controller 250 may use optical flow analysis to determine alocation of the end effector tool 512 with respect to the work surface520. For example, optical flow analysis may be used to determinemovement of the camera 610 with respect to the work surface 520. Sincethe camera 610 and the end effector tool 512 may be adjacent to eachother on the robotic arm 514, the movement of the camera 610 may beapproximately the same as the movement of the end effector tool 512. Thecontroller 250 may include program instructions for determining, basedon the determined movement of the camera 610, a corresponding change incoordinates for the end effector tool 512 within the worksite coordinateframe 100. Location data indicating the new coordinates of the endeffector tool 512 may be stored in the data storage 254 of thecontroller 250.

In order for the local location measurement system 600 to providelocation data indicating the location of the end effector tool 512 at ahigher refresh rate than the global location measurement system 500, thecamera 610 may be a high speed camera. For example, the camera 610 maycapture video of the work surface 520 at a frame rate of 250 frames persecond or higher.

Many techniques for determining optical flow are well known to thoseskilled in the art. Such techniques include (i) phase correlation, (ii)block-based methods, (iii) differential methods such as the Lucas-Kanadeoptical flow method, the Horn-Schunk method, the Buxton-Buxton method,and the Black Jepson method, and (iv) discrete optimization methods,among others.

To determine optical flow at a given point in time, the controller 250may apply the Lucas-Kanade method or any other appropriate technique tocompare the optical flow between two video frames from the camera 610.In an exemplary embodiment, the determined optical flow may take theform of optical-flow data that is stored in the data storage 254. For agiven optical-flow determination, the resulting optical-flow data mayinclude a set of sample points that are identified by their coordinateswithin the video frame, and for each sample point, data indicating themagnitude and direction of the corresponding optical-flow vector at thesample point. Visually, the optical flow determined by the controller250 may be two-dimensionally represented as an “optical-flow field,”which shows the set of sample points within the video frame and theoptical-flow vectors that indicate the respective optical flow at eachsample point.

FIG. 7 depicts an illustration of an optical flow field 700 according toan example embodiment. As shown, the optical flow field 700 includessample points throughout the frame, with an optical flow vector thatcorresponds to each sample point indicating the magnitude and directionof the optical flow that is determined at the sample point.

Generally, each sample point in the optical flow field 700 correspondsto a certain pixel in the video frame from which the optical flow isderived. Therefore, the range of coordinate values for the samplespoints may be defined by the resolution of the video for which theoptical flow is being determined. For instance, the example optical-flowfield 700 provides the optical flow for video with a resolution of 640pixels by 480 pixels (“640×480”). As such, each sample point in theoptical flow field 700 may be identified by its respective coordinateswithin the 640×480 video frame. In the illustrated example, forinstance, the sample points are arranged in an 11×15 grid, withcoordinates ranging from [40, 40] at sample point A in the lower leftcorner, to [600, 440] in the upper right corner.

It should be understood that the number and arrangement of sample pointsin an optical flow field may vary as a matter of engineering designchoice. For example, while an exemplary controller 250 may be configuredto determine optical flow for the same number of sample points as shownin the optical flow field 700, the number of sample points may beincreased to help increase the accuracy of the optical flow.Furthermore, while sample points in the optical flow field 700 are shownas being arranged in a grid and spaced by 40 pixels, the arrangement andspacing of sample points for which optical flow is determined may varyas a matter of engineering design choice. Yet further, the number ofpoints for which an optical-flow vector is determined may vary dependingon the particular technique for determining optical flow that isemployed.

Referring back to FIG. 2, an exemplary controller 250 may be configuredto receive a live video feed from local sensor(s) 230 (e.g., the camera610 depicted in FIG. 6) and continually update and analyze the opticalflow as frames of the video feed are received. As such, the controller250 may continually monitor the optical flow to determine movement ofthe end effector tool 220 with respect to a work surface and storeupdated coordinates in the data storage 254 indicating the location ofthe end effector tool 220 within a worksite coordinate frame.

Employing optical flow analysis to determine a current location of theend effector tool 512 relies on dead reckoning techniques (i.e., theprocess of calculating a current position by using a previouslydetermined position and advancing the previous position based upon knownor estimated speeds over time and course). Dead reckoning is subject toerror accumulation, and the local location measurement system 600 istherefore subject to error accumulation. On the other hand, the globallocation measurement system 500 measures the location of the endeffector tool 512 with respect to the worksite coordinate frame ratherthan through dead reckoning. Thus, the global location measurementsystem 500 may be combined with the local measurement system 600 toremove the accumulated error from dead reckoning.

Referring back to FIG. 6, since optical flow analysis only providestwo-dimensional movement information, the local location measurementsystem 600 may further include a distance sensor 612 to provide a thirddimension of movement information. As depicted in FIG. 6, the distancesensor 612 may be a laser distance sensor. However, the distance sensor612 may alternatively be any distance sensor configured to measure thedistance between the distance sensor 612 and the work surface 520.

Since the optical flow analysis provides movement vectors for thetwo-dimensional axes formed by the field of view of the camera 610, thedistance sensor 612 may be configured to measure the distance betweenthe distance sensor 612 and the work surface 520 along an axisperpendicular to the field of view of the camera 610. In order toincrease the accuracy of the distance sensor measurement, the distancesensor 612 may be mounted adjacent to the camera 610.

Distance data indicating the distance between the distance sensor 612and the work surface 520 may be communicated to the controller 250. Thecontroller 250 may include program instructions for determining, basedon the determined distance data, a corresponding change in coordinatesfor the end effector tool 512 within the worksite coordinate frame 100.Location data indicating the current coordinates of the end effectortool 512 may be stored in the data storage 254 of the controller 250.

In some embodiments, the distance data may further be used to focus thecamera 610. When the end effector tool 512 is performing a task on thework surface 520, the camera 610 may be moved closer to and farther fromthe work surface. Changing the distance between the camera 610 and thework surface 520 may cause the camera 610 to become unfocused. Toaccount for this, the controller 250 may include program instructionsfor configuring the focal length of the camera 610 based on the distancedata received from the distance sensor 612. For example, if the distancedata indicates the camera 610 has moved closer to the work surface 520,the controller 250 may communicate instructions to the camera 610 toreduce the focal length of the camera 610 accordingly.

The local location measurement system 600 may further include a motionsensor 620. As depicted in FIG. 6, the motion sensor 620 may be mountedon the end effector tool 512. The motion sensor 620 may be any sensorconfigured to measure local movement of the end effector tool 512 at ahigh frequency. For example, the motion sensor 620 may be an IMUconfigured to provide data at a refresh rate of 1 kHz or higher.Further, while an IMU may provide data at a higher refresh rate than theoptical flow analysis, an IMU may have a lower spatial accuracy due toerror accumulation and lower precision sensors.

As is understood by those skilled in the art, an IMU may measure abody's specific force and angular rate using a combination ofaccelerometers and gyroscopes. Based on the specific force and angularrate measurements from the IMU, the controller 250 may include programinstructions for determining a location of the IMU, and, consequently,the end effector tool 512, through dead reckoning. For example, knowncoordinates of the end effector tool 512 may be stored in the datastorage 254. The known coordinates may be determined based on data fromthe global location measurement system 500 or from the camera 610 anddistance sensor 612 in the local location measurement system 600. TheIMU may provide data indicating the specific force and angular rate ofthe end effector tool 512. Based on the known coordinates and thespecific force and angular rate of the end effector tool 512, thecontroller may determine current coordinates indicating a position ofthe end effector tool 512 within the worksite coordinate frame 100.Location data indicating the current coordinates of the end effectortool 512 may be stored in the data storage 254 of the controller 250.

Similar to the optical flow analysis, the dead reckoning process usingdata from the IMU is subject to error accumulation. The IMU may providedata at a higher refresh rate than the camera 610 (e.g., 1 kHz versus250 Hz), and may therefore result in a larger error accumulation.Determining the current coordinates of the end effector tool 512 usingoptical flow analysis, due to its lower refresh rate and higher spatialaccuracy, may remove some of the accumulated error resulting frommeasurements based on data from the IMU. Similarly, determining thecurrent coordinates of the end effector tool 512 using data from theglobal location measurement system 500, due to the lower refresh rateand higher spatial accuracy of the global sensors 240, may remove theaccumulated error resulting from measurements based on data from thelocal location measurement system 600.

Data provided by local sensor(s) 230, such as the motion sensor 620 orthe camera 610, is not only at risk of error accumulation, but is alsosubject to error caused by drift. In example embodiments, drift occurswhen the output of the local sensor(s) 230 changes over time independentfrom their location measurements. Drift may be caused by changes intemperature, electronics stabilizing, electronics aging, and/orimperfections in the local sensor(s) 230, among other causes.

FIG. 8 depicts a flowchart of a method 800 according to an exampleembodiment. The method 800 may be implemented to determine an unintendedmovement of an end effector tool of a robotic device and stabilize theend effector tool based on the determined unintended movement. Theunintended movement of the end effector tool may be any movement of theend effector tool that is not caused by a controller of the roboticdevice. For example, the unintended movement may be due to vibrations ofthe robotic device, the robotic device sinking into the ground, or anyother unintentional movement of the end effector tool.

Step 802 involves providing an end effector tool mounted to a moveablecomponent of a robotic device, where the end effector tool is configuredto perform a task on a work surface in a worksite coordinate frame. Therobotic device may be the robotic device 300 depicted in FIG. 3, or,more specifically, the robotic device may be the worker robot 130depicted in FIG. 1. The worksite coordinate frame may be the worksitecoordinate frame 100 depicted in FIG. 1.

Step 804 involves receiving first location data of the end effector toolwith respect to a work surface. The first location data may be providedby a local location measurement system, such as the local locationmeasurement system 600 depicted in FIG. 6. The first location data maybe provided at a high refresh rate (e.g., 1 kHz or higher), but may havea low spatial accuracy due to error accumulation and/or low precisionsensors. For example, the first location data may be data indicating thespecific force and angular rate of the end effector tool, and it may beprovided by an IMU, such as the motion sensor 620 depicted in FIG. 6.

Step 806 involves determining, based on the first location data, anunintended movement of the end effector tool with respect to the worksurface. For example, the unintended movement may be determined throughdead reckoning. Based on a known previous location and the dataindicating the specific force and angular rate of the end effector tool,the controller 250 may determine a current location and velocity of theend effector tool.

Step 808 involves receiving image data indicating the work surface inthe worksite coordinate frame. The image data may comprise a videorecording of the work surface and it may be provided by a camera, suchas the camera 610 depicted in FIG. 6. The image data may be provided ata lower refresh rate (e.g., 250 Hz) than the first location data, butmay have a higher spatial accuracy than the first location data becauseit may not be subject to as much error accumulation.

Step 810 involves determining, based on the image data, an unintendedmovement of the end effector tool with respect to the work surface. Forexample, the unintended movement may be determined through optical flowanalysis. The controller 250 may perform optical flow analysis todetermine the unintended movement of the end effector tool with respectto the work surface.

Step 812 involves receiving second location data indicating a secondlocation of the end effector tool with respect to the worksitecoordinate frame. The second location data may be provided by the globallocation measurement system 500. The second location data may beprovided at a lower refresh rate (e.g., approximately 1 Hz) than theimage data but may have a higher spatial accuracy than the image data,because it is not subject to error accumulation and the sensor may havea much higher precision. For example, the second location data may beprovided by a laser tracker system and may include data indicating thelocation of a marker on the end effector tool with respect to the lasertracker.

Step 814 involves determining, based on the second location data, anunintended movement of the end effector tool with respect to theworksite coordinate frame. For example, the controller 250 may storelocation data from the laser tracker in the data storage 254. Thelocation data from the laser tracker may include the second locationdata as well as previous location data from the laser tracker indicatinga prior location of the end effector tool. By determining the differencebetween the previous location data and the second location data, thecontroller 250 may determine an unintended movement of the end effectortool with respect to the worksite coordinate frame.

Step 816 involves instructing the robotic device to manipulate the endeffector tool to counteract the unintended movements of the end effectortool with respect to the work surface and worksite coordinate frame. Thecontroller 250 may communicate instructions to the robotic device tocounteract the unintended movements determined based on the firstlocation data, image data, and second location data. The determinedunintended movement based on the first location data may be representedby a first vector with a magnitude and direction. In order to counteractthis movement, the controller 250 may communicate first instructions tothe robotic device to move the end effector tool in a manner havingopposite magnitude and direction as the first vector. The firstinstructions may be provided at the same refresh rate as the firstlocation data (e.g., 1 kHz or higher).

Similarly, the determined unintended movement based on the image datamay be represented by a second vector with a magnitude and direction.The controller 250 may communicate second instructions to the roboticdevice to move the end effector tool in a manner having oppositemagnitude and direction as the second vector. The second instructionsmay be provided at the same refresh rate as the image data (e.g., 250Hz).

And the determined unintended movement based on the second location datamay be represented by a third vector with a magnitude and direction. Thecontroller 250 may communicate third instructions to the robotic deviceto move the end effector tool in a manner having opposite magnitude anddirection as the third vector. The third instructions may be provided atthe same refresh rate as the second location data (e.g., approximately 1Hz).

By instructing the robotic device to move the end effector tool in anopposite manner as the determined unintended movements based on thefirst location data, image data, and second location data, the method800 allows for stabilizing the end effector tool at a high rate (e.g., 1kHz or higher) while providing more accurate stabilizing instructions ata lower rate (e.g., 250 Hz) to reduce cumulative measurement errors, andproviding even higher accuracy stabilizing instructions at an even lowerrate (e.g., 1 Hz) to remove cumulative measurement errors and accountfor lower resolution of the first location data and image data.

III. CONCLUSION

The particular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other embodiments may includemore or less of each element shown in a given Figure. Further, some ofthe illustrated elements may be combined or omitted. Yet further, anexemplary embodiment may include elements that are not illustrated inthe Figures.

Additionally, while various aspects and embodiments have been disclosedherein, other aspects and embodiments will be apparent to those skilledin the art. The various aspects and embodiments disclosed herein are forpurposes of illustration and are not intended to be limiting, with thetrue scope and spirit being indicated by the following claims. Otherembodiments may be utilized, and other changes may be made, withoutdeparting from the spirit or scope of the subject matter presentedherein. It will be readily understood that the aspects of the presentdisclosure, as generally described herein, and illustrated in thefigures, can be arranged, substituted, combined, separated, and designedin a wide variety of different configurations, all of which arecontemplated herein.

What is claimed is:
 1. A method comprising: providing an end effectortool mounted to a moveable component of a robotic device, wherein theend effector tool is configured to perform a task on a work surface in aworksite coordinate frame; receiving, from a first sensor, image dataindicative of the work surface; determining, based on the image data,first movement information indicating unintended movement over time ofthe end effector tool with respect to the work surface; receiving, froma second sensor, location data indicating a location of the end effectortool with respect to the worksite coordinate frame; determining, basedon the location data, second movement information indicating unintendedmovement over time of the end effector tool with respect to the worksitecoordinate frame; determining, based on the first and second movementinformation, one or more adjustments to one or more control signals forthe robotic device in order to counteract the unintended movements ofthe end effector tool with respect to the work surface and worksitecoordinate frame; and communicating instructions to the robotic deviceto operate in accordance with the one or more adjustments.
 2. The methodof claim 1, further comprising: receiving, from a third sensor, secondlocation data indicating a second location of the end effector tool withrespect to the work surface; determining, based on the second locationdata, third movement information indicating unintended movement overtime of the end effector tool with respect to the work surface;determining, based on the third movement information, one or moreadjustments to one or more control signals for the robotic device inorder to counteract the unintended movement over time of the endeffector tool with respect to the work surface; and communicatinginstructions to the robotic device to operate in accordance with the oneor more adjustments.
 3. The method of claim 1, wherein the first sensorcomprises an optical flow sensor.
 4. The method of claim 3, furthercomprising receiving, from a laser sensor, distance data indicative of adistance of the work surface from the laser sensor.
 5. The method ofclaim 4, further comprising adjusting the focal length of the opticalflow sensor based on the distance data received from the laser sensor.6. The method of claim 4, further comprising: determining, based on thedistance data received from the laser sensor, fourth movementinformation indicating unintended movement over time of the end effectortool with respect to the work surface; determining, based on the fourthmovement information, one or more adjustments to one or more controlsignals for the robotic device in order to counteract the unintendedmovement of the end effector tool; and communicating instructions to therobotic device to operate in accordance with the one or moreadjustments.
 7. The method of claim 1, wherein the second sensorcomprises a motion capture sensor, wherein a plurality of motion capturemarkers are positioned in the work environment, and wherein at least oneof the plurality of motion capture markers is attached to the endeffector tool.
 8. The method of claim 1, wherein the second sensorcomprises a laser tracker configured to measure a location of one ormore markers attached to the robotic device, wherein the location of theend effector tool is determined with respect to the location of the oneor more markers attached to the robotic device.
 9. The method of claim8, wherein at least one of the one or more markers is attached to theend effector tool, wherein the end effector tool performing a task onthe work surface interrupts a line of sight between the laser trackerand the at least one of the one or more markers attached to the endeffector tool, wherein the robotic device is configured to periodicallymanipulate the end effector tool to restore the line of sight.
 10. Themethod of claim 2, wherein the third sensor comprises an inertialmeasurement unit.
 11. A system comprising: an end effector tool mountedto a moveable component of a robotic device, wherein the end effectortool is configured to perform a task on a work surface in a worksitecoordinate frame; and a control system configured to: receive, from afirst sensor, image data indicative of the work surface; determine,based on the image data, first movement information indicatingunintended movement over time of the end effector tool with respect tothe work surface; receive, from a second sensor, location dataindicating a location of the end effector tool with respect to theworksite coordinate frame; determine, based on the location data, secondmovement information indicating unintended movement over time of the endeffector tool with respect to the worksite coordinate frame; determine,based on the first and second movement information, one or moreadjustments to one or more control signals for the robotic device inorder to counteract the unintended movements of the end effector toolwith respect to the work surface and worksite coordinate frame; andcommunicate instructions to the robotic device to operate in accordancewith the one or more adjustments.
 12. The system of claim 11, whereinthe control system is further configured to: receive, from a thirdsensor, second location data indicating a second location of the endeffector tool with respect to the work surface; determine, based on thesecond location data, third movement information indicating unintendedmovement over time of the end effector tool with respect to the worksurface; determine, based on the third movement information, one or moreadjustments to one or more control signals for the robotic device inorder to counteract the unintended movement over time of the endeffector tool with respect to the work surface; and communicateinstructions to the robotic device to operate in accordance with the oneor more adjustments.
 13. The system of claim 11, wherein the firstsensor comprises an optical flow sensor.
 14. The system of claim 13,wherein the control system is further configured to receive, from alaser sensor, distance data indicative of a distance of the work surfacefrom the laser sensor.
 15. The system of claim 14, wherein the controlsystem is further configured to adjust the focal length of the opticalflow sensor based on the distance data received from the laser sensor.16. The system of claim 14, wherein the control system is furtherconfigured to: determine, based on the distance data received from thelaser sensor, fourth movement information indicating unintended movementover time of the end effector tool with respect to the work surface;determine, based on the fourth movement information, one or moreadjustments to one or more control signals for the robotic device inorder to counteract the unintended movement of the end effector tool;and communicate instructions to the robotic device to operate inaccordance with the one or more adjustments.
 17. The system of claim 11,wherein the second sensor comprises a motion capture sensor, wherein aplurality of motion capture markers are positioned in the workenvironment, and wherein at least one of the plurality of motion capturemarkers is attached to the end effector tool.
 18. The system of claim11, wherein the second sensor comprises a laser tracker configured tomeasure a location of one or more markers attached to the roboticdevice, wherein the location of the end effector tool is determined withrespect to the location of the one or more markers attached to therobotic device.
 19. The system of claim 18, wherein at least one of theone or more markers is attached to the end effector tool, wherein theend effector tool performing a task on the work surface interrupts aline of sight between the laser tracker and the at least one of the oneor more markers attached to the end effector tool, wherein the roboticdevice is configured to periodically manipulate the end effector tool torestore the line of sight.
 20. The system of claim 12, wherein the thirdsensor comprises an inertial measurement unit.